feat: pre-messages / next version of download on demand by Simon-Laux · Pull Request #7371 · chatmail/core

Simon-Laux · 2025-10-29T20:53:40Z

feat: Remove "Download maximum available until" and remove stock string DC_STR_DOWNLOAD_AVAILABILITY #7369
remove: partial downloads (remove creation of the stub messages) #7373
feat: pre messages can now get read receipts #7433
feat: send pre-message on messages with large attachments #7410
feat: receive pre-messages and adapt download on demand #7431
... for now read the issue for more info on todo items and what this is about

This is the branch for #7367
closes #7367

Currently removed tests

test_something.py::test_reaction_to_partially_fetched_msg
python/tests/test_1_online.py::test_webxdc_download_on_demand
src/calls/calls_tests.py::test_no_partial_calls
src/message/message_tests.rs::test_markseen_not_downloaded_msg
src/receive_imf/receive_imf_tests.rs::test_download_later
src/receive_imf/receive_imf_tests.rs::test_create_group_with_big_msg
src/receive_imf/receive_imf_tests.rs::test_prefer_references_to_downloaded_msgs
src/receive_imf/receive_imf_tests.rs::test_partial_download_key_contact_lookup
src/webxdc/webxdc_tests.rs::test_webxdc_update_for_not_downloaded_instance
src/download.rs::test_download_limit
src/download.rs::test_partial_receive_imf
src/download.rs::test_partial_download_and_ephemeral
src/download.rs::test_status_update_expands_to_nothing
src/download.rs::test_mdn_expands_to_nothing
src/download.rs::test_partial_download_trashed
src/reaction.rs::test_partial_download_and_reaction

Progress of the tests

Overview about the recycled(♻️) and dropped(🗑️) tests of the tests that I removed in #7373

🗑️test_reaction_to_partially_fetched_msg
🗑️ test_partial_download_and_ephemeral
🗑️ test_status_update_expands_to_nothing
♻️ test_partial_download_trashed
♻️ test_partial_download_and_reaction
♻️ test_partial_group_consistency
🗑️ test_partial_download_key_contact_lookup
♻️ test_webxdc_update_for_not_downloaded_instance
🗑️test_no_partial_calls
♻️ test_markseen_not_downloaded_msg
♻️ test_msg_seen_on_imap_when_downloaded
♻️ test_create_group_with_big_msg
♻️ test_download_later
🗑️ test_prefer_references_to_downloaded_msgs
♻️test_download_on_demand
♻️test_download_limit_chat_assignment
🗑️test_webxdc_download_on_demand

All the recycled tests were already recycled/re-made except for some which are still to do:
(none)

Furthermore there need to be new tests to test the downloading/scheduling changes.

TODO Tests

sending pre-message, post-message
metadata in pre-message
process receiving
display attachment info in pre-message
test scheduler imap fetching
test failing download sets state to failed
multi device tests
deletion tests, incl. sender-side deletion, whole chat deletion

postponed to do tests

(none)

discarded/ignored test ideas

maybe we should reconsider / discuss those?

test that full message, replacing pre-message does not work if the message encryption state is not the same for both messages (both full- and pre-message)
- unencrypted pre- and full-message are not sent

Known bugs:

feat: pre-messages / next version of download on demand #7371 (comment)

src/sql/migrations.rs

deltachat-ffi/deltachat.h

deltachat-jsonrpc/src/api/types/message.rs

src/chat.rs

src/download.rs

src/chat.rs

src/download/pre_msg_metadata.rs

src/scheduler.rs

src/sql/migrations.rs

src/stock_str.rs

src/tests.rs

src/imap.rs

src/download.rs

src/message.rs

It currently fails because pre-messages aren't deleted on IMAP.

Hyphen shouldn't be surrounded by spaces in general.

…th pre-message ones When inserting pre-message checks in the middle, if they evaluate to false, the remaining checks aren't performed. While this may be correct currently, it's hard to maintain. Better move the pre-message checks into a separate block.

…_pre_message() Now it goes before `scan_folders()`. Scanning folders isn't that important, it's even debounced to once per minute. Before, `download_msgs()` even preceded the whole `fetch_idle()`.

…s as seen on IMAP

It's the post-message metadata by its nature and it's even put into the Chat-Post-Message-Metadata header.

…re-messages

This shouldn't cause message reordering issues because the UIs don't allow to set too small DownloadLimit, but this will help download small, e.g. chat messages first and only then big emails. Even if we decide to revert this suggestion, the test is still useful.

It never worked as expected: if messages showed up on IMAP later, they weren't deleted. Messages must be trashed with the `deleted` flag set, not just deleted from the db.

I.e. if DeleteServerAfter is set or BccSelf is unset (single-device mode) for chatmail. Looks like this never worked as expected.

link2xt

I will merge this as one commit after tagging 2.37.0. Any additional changes can be separate PRs on top.

iequidoo · 2026-01-09T01:08:45Z

There's no good commit message however. I think in such cases we need to copy the description from the issue, it's good enough here ("Motivation" and "Implementation" sections look enough). Otherwise w/o access to github it will be hard to understand many things. I'd consider amending the commit and force-pushing main

Simon-Laux changed the title ~~[Branch] Pre-messages / next version of download on demand~~ [Branch] pre-messages / next version of download on demand Oct 29, 2025

Simon-Laux force-pushed the pre-messages branch from 6fe6723 to 8140ebc Compare November 6, 2025 13:13

link2xt force-pushed the pre-messages branch 3 times, most recently from 2f1c383 to 6bcc795 Compare December 11, 2025 12:00

link2xt mentioned this pull request Dec 11, 2025

feat: receive pre-messages and adapt download on demand #7431

Merged

26 tasks

link2xt force-pushed the pre-messages branch 6 times, most recently from 8443b17 to 702771f Compare December 11, 2025 21:24

Hocuri reviewed Dec 11, 2025

View reviewed changes

src/sql/migrations.rs Outdated Show resolved Hide resolved

link2xt force-pushed the pre-messages branch 2 times, most recently from bb0fd7c to a98fe05 Compare December 14, 2025 17:54

link2xt changed the title ~~[Branch] pre-messages / next version of download on demand~~ feat: pre-messages / next version of download on demand Dec 14, 2025

link2xt force-pushed the main branch from e0ae864 to 1ea8ed6 Compare December 14, 2025 19:44

Hocuri reviewed Dec 18, 2025

View reviewed changes

src/sql/migrations.rs Show resolved Hide resolved

iequidoo reviewed Dec 22, 2025

View reviewed changes

iequidoo reviewed Dec 24, 2025

View reviewed changes

iequidoo reviewed Dec 27, 2025

View reviewed changes

src/imap.rs Outdated Show resolved Hide resolved

iequidoo reviewed Dec 28, 2025

View reviewed changes

src/download.rs Outdated Show resolved Hide resolved

src/download.rs Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

iequidoo force-pushed the pre-messages branch 2 times, most recently from 4a119fc to 84ce946 Compare December 30, 2025 04:30

This comment was marked as resolved.

Sign in to view

iequidoo reviewed Dec 30, 2025

View reviewed changes

src/message.rs Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

iequidoo force-pushed the pre-messages branch 2 times, most recently from 18346a1 to 77ab447 Compare January 2, 2026 20:01

iequidoo and others added 23 commits January 8, 2026 12:22

test_msg_text_on_lost_pre_msg

b082e75

Add test_pre_and_post_msgs_deleted

eab95b7

It currently fails because pre-messages aren't deleted on IMAP.

Add test_post_msg_bad_sender

801c0c6

run ruff check --fix

0cbfa5d

Add test_lost_pre_msg_vs_new_member

33a3057

fix: Delete pre-messages on IMAP

0fe055b

Make {download,available_post_msgs}.rfc724_mid columns PRIMARY KEY

e9f4f11

Replace '-' with en dash for file size separation

f0125fe

Hyphen shouldn't be surrounded by spaces in general.

Make PreMessageMode usable without Option

a00efb6

Fix python lint

668d05c

Move download_msgs() right after download_known_post_messages_without…

300d48e

…_pre_message() Now it goes before `scan_folders()`. Scanning folders isn't that important, it's even debounced to once per minute. Before, `download_msgs()` even preceded the whole `fetch_idle()`.

fix: prefetch_should_download: Mark already existing outgoing message…

be5e403

…s as seen on IMAP

Rename PreMsgMetadata to PostMsgMetadata

f6bb1b1

It's the post-message metadata by its nature and it's even put into the Chat-Post-Message-Metadata header.

Improve download_limit docs

96a0757

test_download_limit_chat_assignment: Work around lost and reordered p…

5599fa2

…re-messages

test_delete_available_msg

d56d940

fix deletion of reordered pre- and post-messages

21f6680

fix pre-message deletion on the sender side

449ac19

fix message deletion on IMAP when deleting the whole chat

7852074

It never worked as expected: if messages showed up on IMAP later, they weren't deleted. Messages must be trashed with the `deleted` flag set, not just deleted from the db.

test_delete_fully_downloaded_msg

1f4cb72

fix: Delete received message on IMAP immediately if this is configured

5d62eb8

I.e. if DeleteServerAfter is set or BccSelf is unset (single-device mode) for chatmail. Looks like this never worked as expected.

iequidoo force-pushed the pre-messages branch from ee91a0b to 5d62eb8 Compare January 8, 2026 15:22

This comment was marked as outdated.

Sign in to view

link2xt approved these changes Jan 8, 2026

View reviewed changes

link2xt merged commit 2631745 into main Jan 8, 2026
30 checks passed

link2xt deleted the pre-messages branch January 8, 2026 22:14

Simon-Laux mentioned this pull request Jan 11, 2026

Not fully downloaded message edit is lost #6806

Closed

Uh oh!

Conversation

Simon-Laux commented Oct 29, 2025 • edited by iequidoo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Currently removed tests

Progress of the tests

TODO Tests

postponed to do tests

discarded/ignored test ideas

Known bugs:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as outdated.

link2xt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iequidoo commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Simon-Laux commented Oct 29, 2025 •

edited by iequidoo

Loading

iequidoo commented Jan 9, 2026 •

edited

Loading